Skip to content

Memtable enhancement for query#14591

Merged
jt2594838 merged 50 commits intomasterfrom
force_ci/split_chunk
Feb 21, 2025
Merged

Memtable enhancement for query#14591
jt2594838 merged 50 commits intomasterfrom
force_ci/split_chunk

Conversation

@HTHou
Copy link
Contributor

@HTHou HTHou commented Dec 30, 2024

Description

  1. Ensure memory data query performance and avoid excessive time-consuming memory data sorting
  2. Reduce frequent triggering of flush, which causes the number of files to grow too fast
  3. Ensure the performance of data queries in the disk and avoid flushing the chunk size to the disk to be too large

@codecov
Copy link

codecov bot commented Dec 30, 2024

Codecov Report

Attention: Patch coverage is 59.84757% with 1001 lines in your changes missing coverage. Please review.

Project coverage is 39.36%. Comparing base (69d9dae) to head (8ff8928).
Report is 83 commits behind head on master.

Files with missing lines Patch % Lines
...e/dataregion/memtable/AlignedWritableMemChunk.java 44.07% 203 Missing ⚠️
...e/dataregion/memtable/AlignedReadOnlyMemChunk.java 29.24% 179 Missing ⚠️
...geengine/dataregion/memtable/WritableMemChunk.java 51.05% 139 Missing ⚠️
...geengine/dataregion/memtable/ReadOnlyMemChunk.java 43.36% 64 Missing ⚠️
...he/iotdb/db/utils/datastructure/AlignedTVList.java 71.68% 47 Missing ⚠️
...region/read/reader/chunk/MemAlignedPageReader.java 34.84% 43 Missing ⚠️
...che/iotdb/db/utils/datastructure/BackwardSort.java 0.00% 33 Missing ⚠️
...ne/dataregion/read/reader/chunk/MemPageReader.java 62.50% 21 Missing ⚠️
...engine/schemaregion/utils/ResourceByPathUtils.java 62.26% 20 Missing ⚠️
...rg/apache/iotdb/db/utils/datastructure/TVList.java 88.05% 19 Missing ⚠️
... and 41 more
Additional details and impacted files
@@             Coverage Diff             @@
##             master   #14591     +/-   ##
===========================================
  Coverage     39.36%   39.36%             
  Complexity      193      193             
===========================================
  Files          4490     4502     +12     
  Lines        286232   287968   +1736     
  Branches      35484    35849    +365     
===========================================
+ Hits         112663   113352    +689     
- Misses       173569   174616   +1047     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

HTHou and others added 5 commits January 21, 2025 10:41
* null bitmap for int tvlist

* update min/max timestamp and sequential part of tvlist during insert

* mutable & immutable tvlists in writable memchunk

* copy-on-write array list

* review comments part 1

* fix unit test errors

* review comments part 2

* push down global time filter

* fix MemPageReaderTest case

* fix memory page offsets error

* synchronized sort & MergeSortTvListIterator bug

* tvlist_sort_threshold config property

* bug fix:
* out of mempage bounds check
* overlapped data error during query

* optimize TVListIterator & MergeSortTvListIterator

* retrofit encode when tvlist_sort_threshold is zero

* delay sort & statistic generation to query execution

* fix: skip deleted data during encode

* aligned time series part

* fix: MemAlignedChunkReader page offset

* performance issue:
* change some list to array
* remember row count in tvlist iterator

* fix: memory chunk reader may read more points than expected in one page

* update chunk & page statistic for aligend memchunk by column

* revert: getAlignedValueForQuery

* fix: * CopyOnWriteArrayList for AlignedTVList bitmaps
* memory control of column access

* refactor: Tim/Quick/Backward TVList

* refactor: synchronized tvlist method: sort, putXXX

* refactor: change list to array in AlignedTVList iterator

* revert: remove CopyOnWriteArrayList

* refactor: clone MergeSort iterator from ReadOnlyChunk

* fix: clone working tvlist during flush if there is query on it

* fix: writable mem chunk flush conditions

* refactor: add annotation and variable/function rename

* fix: * remove delete method in BinaryTVList
* filter deleted data in WritableMemChunk encode

* fix: remove getSortedTvListForQuery in SeriesRegionScan

* fix: TsFileProcessorTest unit test

* fix: IoTDBNullIdQueryIT.noMeasurementColumnsSelectTest

* fix: delete column of aligned time series

* fix: aligned timeseries encode bug

* fix: IoTDBGroupByNaturalMonthIT

* remove avgSeriesPointNumberThreshold setting

* fix: IoTDBDeleteAlignedTimeseriesIT & AlignedTVListTest

* fix: Copy globalTimeFilter due to GroupByMonthFilter

* reset tmpLength for backward sort

* * fix TVList clear
* bitmap mark
* sequence row count

* hot-load TVLIST_SORT_THRESHOLD

* fix: isNullValue caller

* fix unit test

* refactor: abstract prepareTvListMapForQuery method

* refactor:  clear/clone/expand indices and bitmap

* merge sort using min heap

* fix: WritableMemChunk deserialize

* feat: add index mem cost for TVList

* fix: hot-load tvlist_sort_threshold setting

* remove needless line in property template
@HTHou HTHou changed the title Remove the logic of memtable flush controlled by memory series size Memtable enhancement for query Feb 7, 2025
@sonarqubecloud
Copy link

sonarqubecloud bot commented Feb 7, 2025

Quality Gate Failed Quality Gate failed

Failed conditions
6.6% Duplication on New Code (required ≤ 5%)
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

@jt2594838 jt2594838 merged commit 8532609 into master Feb 21, 2025
70 of 71 checks passed
HTHou added a commit that referenced this pull request Feb 25, 2025
* Split non_aligned charge text chunk

* dev non_aligned

* dev aligned chunk split

* new type

* dev aligned binary chunk split

* Fix binary size calculatation

* fix IT

* update IoTDBDuplicateTimeIT.java

* fix pipe IT

* change method names

* add ut

* add UT

* remove useless methods

* fix UT

* fix /FileReaderManagerTest

* fix win UT

* add binary test

* Add Aligned UTs

* fix win ut

* improve coverage

* fix comments

* fix windows UT

* fix review

* fix review

* fix review

* target chunk size count non binary

* fix compile

* fix UT

* Tvlist feat new (#14616)

* null bitmap for int tvlist

* update min/max timestamp and sequential part of tvlist during insert

* mutable & immutable tvlists in writable memchunk

* copy-on-write array list

* review comments part 1

* fix unit test errors

* review comments part 2

* push down global time filter

* fix MemPageReaderTest case

* fix memory page offsets error

* synchronized sort & MergeSortTvListIterator bug

* tvlist_sort_threshold config property

* bug fix:
* out of mempage bounds check
* overlapped data error during query

* optimize TVListIterator & MergeSortTvListIterator

* retrofit encode when tvlist_sort_threshold is zero

* delay sort & statistic generation to query execution

* fix: skip deleted data during encode

* aligned time series part

* fix: MemAlignedChunkReader page offset

* performance issue:
* change some list to array
* remember row count in tvlist iterator

* fix: memory chunk reader may read more points than expected in one page

* update chunk & page statistic for aligend memchunk by column

* revert: getAlignedValueForQuery

* fix: * CopyOnWriteArrayList for AlignedTVList bitmaps
* memory control of column access

* refactor: Tim/Quick/Backward TVList

* refactor: synchronized tvlist method: sort, putXXX

* refactor: change list to array in AlignedTVList iterator

* revert: remove CopyOnWriteArrayList

* refactor: clone MergeSort iterator from ReadOnlyChunk

* fix: clone working tvlist during flush if there is query on it

* fix: writable mem chunk flush conditions

* refactor: add annotation and variable/function rename

* fix: * remove delete method in BinaryTVList
* filter deleted data in WritableMemChunk encode

* fix: remove getSortedTvListForQuery in SeriesRegionScan

* fix: TsFileProcessorTest unit test

* fix: IoTDBNullIdQueryIT.noMeasurementColumnsSelectTest

* fix: delete column of aligned time series

* fix: aligned timeseries encode bug

* fix: IoTDBGroupByNaturalMonthIT

* remove avgSeriesPointNumberThreshold setting

* fix: IoTDBDeleteAlignedTimeseriesIT & AlignedTVListTest

* fix: Copy globalTimeFilter due to GroupByMonthFilter

* reset tmpLength for backward sort

* * fix TVList clear
* bitmap mark
* sequence row count

* hot-load TVLIST_SORT_THRESHOLD

* fix: isNullValue caller

* fix unit test

* refactor: abstract prepareTvListMapForQuery method

* refactor:  clear/clone/expand indices and bitmap

* merge sort using min heap

* fix: WritableMemChunk deserialize

* feat: add index mem cost for TVList

* fix: hot-load tvlist_sort_threshold setting

* remove needless line in property template

---------

Co-authored-by: shizy <shizy04@gmail.com>
@HTHou HTHou deleted the force_ci/split_chunk branch February 26, 2025 03:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants